POC: gate RayService zero-downtime upgrade with workload slicing by kevin85421 · Pull Request #11264 · kubernetes-sigs/kueue

kevin85421 · 2026-05-17T22:11:26Z

Summary

POC for #11102. Prevents the pending RayCluster created during a zero-downtime RayService upgrade from running before Kueue admits a workload slice that covers its quota demand.

Depends on ray-project/kuberay#4841 (top-level RayService.Spec.Suspend + nested template-suspend semantics). The vendored rayservice_types.go is synced from a local KubeRay checkout with that PR applied.

For a step-by-step walkthrough of how the two suspend fields interact across creation, admission, and a zero-downtime upgrade, see this comment on ray-project/kuberay#4841.

Design

Action	Owner
Child RayCluster born suspended	KubeRay reads `RayService.Spec.RayClusterSpec.Suspend=true` template gate at creation
Persistent template gate	Kueue's `Suspend()` sets nested `Suspend=true`; `RunWithPodSetsInfo()` deliberately leaves it true
Quota for active + pending during upgrade	`PodSets()` lists live children via the `ray.io/originated-from-cr-{name,crd}` labels (mirrors KubeRay's `RayServiceRayClustersAssociationOptions`), unions PodSets by name and sums counts. Same keys across the 1↔2 child transitions so `EnsureWorkloadSlices` handles upgrade as scale-up and post-upgrade as scale-down
Release the gate	`Reconcile` post-step `unsuspendAdmittedChildren` patches each child RayCluster's `Spec.Suspend=false` (not the RayService's) once the latest workload slice is admitted; a race-guard checks the slice's PodSet counts cover the current children's required counts before patching, so the pending child stays gated while the new slice is still pending
Preemption / total stop	`Suspend()` also sets top-level `Spec.Suspend=true`; KubeRay deletes all owned resources

Prerequisites for testing

Feature gate ElasticJobsViaWorkloadSlices=true on the Kueue manager.
kueue.x-k8s.io/elastic-job: "true" annotation on the RayService.
KubeRay built with [RayService][Kueue] Support top-level Spec.Suspend for zero-downtime upgrade ray-project/kuberay#4841.

Known limitations

Heterogeneous PodSpec on the same group name: PodSets() keeps the first child's template, so resource-changing upgrades under-/over-account quota. Same-resource upgrades (rayVersion / image / env) are exact.
No webhook validation: a user can manually clear Spec.RayClusterSpec.Suspend, breaking the gate.
MultiKueue adapter not updated: rayservice_multikueue_adapter.go hasn't been taught the new suspend semantics.

Test plan

Automated:

Unit tests for PodSets() across bootstrap / steady-state / zero-downtime-upgrade child topologies (pkg/controller/jobs/rayservice/rayservice_controller_test.go).
E2E test/e2e/singlecluster/extended/kuberay_test.go — "Should gate a zero-downtime upgrade's pending RayCluster on queue quota": a 3-CPU ClusterQueue gates the pending RayCluster during an upgrade, releasing quota admits the upgrade slice and unsuspends the pending child, the old RayCluster is then deleted and quota settles back to a single cluster's reservation.

Manual (verified locally on kind + KubeRay master with #4841 applied):

Happy path: suspend/resume and upgrade gating end to end.
Limitation 1: resource-changing upgrade demonstrates quota miscount.
Limitation 2: manually clearing the template gate lets the pending pod run un-admitted.
(Out of scope) MultiKueue path.

AI disclosure: this PR was developed with assistance from Claude Code; all changes were reviewed by the author.

k8s-ci-robot · 2026-05-17T22:11:29Z

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2026-05-17T22:11:32Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`1828096`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/6a2f3e20c6883a0008ff5c14

linux-foundation-easycla · 2026-05-17T22:11:37Z

The committers listed above are authorized under a signed CLA.

✅ login: kevin85421 / name: Kai-Hsun Chen (578c392, 7cf656b, bdd98c2, bedca3f, d2415f4)

k8s-ci-robot · 2026-05-17T22:11:37Z

Hi @kevin85421. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

kevin85421 · 2026-05-17T22:18:10Z

 type ClusterUpgradeOptions struct {
 	// The capacity of serve requests the upgraded cluster should scale to handle each interval.
 	// Defaults to 100%.
 	// +kubebuilder:default:=100


ray-project/kuberay#4841 hasn't been merged right now.

kevin85421 · 2026-05-18T04:28:33Z

+	// and the post-upgrade tear-down as a scale-down, without falling back to the
+	// non-slice path.
+	//
+	// POC limitation: when two children share a group name with different PodSpecs


coderabbitai · 2026-06-07T01:21:11Z

Important

Review skipped

Auto reviews are limited based on label configuration.

🚫 Review skipped — only excluded labels are configured. (3)

needs-ok-to-test
do-not-merge/work-in-progress
cncf-cla: no

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: fb5a9ae1-1f68-4014-ad45-e5f3612138ee

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Switch RayService suspend semantics to KubeRay PR kubernetes-sigs#4841's top-level Spec.Suspend (Kueue's stop switch) while keeping the nested RayClusterSpec.Suspend=true as a persistent template gate, so any child RayCluster KubeRay creates -- including the pending one during a zero-downtime upgrade -- is born suspended. Build PodSets from the live child RayClusters (union by group name, sum counts) so the workload's quota reservation reflects active+pending during the upgrade and routes through EnsureWorkloadSlices as a scale-up/scale-down. Keys stay stable across the 1<->2 transition so the slice chain is preserved. Add a Reconcile post-step that unsuspends child RayClusters once the matching workload slice is admitted, with a race-guard that verifies the admitted slice's PodSet counts already cover the union of children's required counts -- prevents prematurely unsuspending the pending child before the upgrade slice is created. Known POC limitations: - Same group name with different PodSpecs across active/pending uses the first child's template, so quota is computed against that template. - MultiKueue adapter does not yet propagate the new suspend semantics. - No webhook validation guarding the persistent RayClusterSpec.Suspend template gate. Vendored rayservice_types.go is synced from a local kuberay checkout with PR kubernetes-sigs#4841 applied; deepcopy is unchanged since Spec.Suspend is a bool value type.

Covers the happy path (1 CPU/2 GiB ClusterQueue gating the upgrade's pending RayCluster) and reproduction recipes for the three known limitations: heterogeneous PodSpec on the same group name, manual tampering with the persistent RayClusterSpec.Suspend gate, and the unfinished MultiKueue adapter.

…g e2e The POC commit changed (*RayService).PodSets and IsSuspended but never updated the unit tests; after the rebase the upstream tests no longer matched the POC's runtime semantics. Unit tests (rayservice_controller_test.go): - TestPodSets: rewrite for the POC's label-based union model. List child RayClusters by KubeRay's association labels and sum PodSet counts across them, instead of upstream's by-name + autoscaling UpdatePodSets model. New cases: bootstrap (no children -> template), single child (reflects the child's live spec), and zero-downtime upgrade (two children -> counts summed: head 1+1=2, group1 2+2=4). Adds a childRayCluster helper and switches the harness to seed multiple labelled children. - TestIsSuspended: assert on the top-level Spec.Suspend (KubeRay kubernetes-sigs#4841) rather than the nested RayClusterSpec.Suspend. E2E test (kuberay_test.go): - Add "Should gate a zero-downtime upgrade's pending RayCluster on queue quota": a 3-CPU ClusterQueue fits one RayCluster (2 CPU) but not the active+pending union (4 CPU). Asserts the pending child is born suspended, the upgrade workload slice reserves both clusters (head count 2) yet stays Pending, the pending child is NOT unsuspended while quota is insufficient, and the gate opens once quota is added. - Find the initial workload by listing rather than by the owner-derived name, since workload slicing adds a slice suffix to the Workload name.

…h merged kuberay#4841 ray-project/kuberay#4841 merged with the same semantics the POC was built against (top-level Spec.Suspend, creation-only nested RayClusterSpec.Suspend, modifyRayCluster preserving the child's Suspend), so no controller changes are needed. - Remove POC-TESTING.md, poc-queues.yaml, ray-service-sample.yaml (manual testing scaffolding, not part of the upstream change). - Align the hand-edited vendored rayservice_types.go with the merged PR: fix the Suspend field comment wording, add the RayServiceResumed condition reason, and drop the unrelated StepSizePercent comment.

… gating e2e Extend the zero-downtime upgrade gating test past the gate-open step: - After promotion, the old RayCluster is deleted (KubeRay's default 60s RayClusterDeletionDelaySeconds) and only the new cluster remains. - The workload slice scales down in place to the single-cluster shape (head=1, 2 pods total), so the ClusterQueue reserves quota for one RayCluster instead of two. Verified locally against a kind cluster running KubeRay master (kuberay#4841 + kubernetes-sigs#4881): 1 Passed | 0 Failed.

kevin85421 · 2026-06-14T23:02:37Z

/easycla

k8s-ci-robot · 2026-06-14T23:02:45Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kevin85421
Once this PR has been reviewed and has the lgtm label, please assign mbobrovskyi for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…elector

kevin85421 · 2026-06-15T18:26:12Z

cc @mimowo KubeRay still doesn't have a release that includes ray-project/kuberay#4841. Should we wait for KubeRay to have a release with the RayService top-level suspend before reviewing/merging this PR? Or can I cut a branch for a KubeRay alpha release, just for Kueue (not for any other users), to unblock this PR?

k8s-ci-robot · 2026-06-20T04:46:00Z

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mimowo · 2026-08-03T19:33:12Z

/ok-to-test
/kind feature
/area integrations
/area elastic-jobs
/area ray-integration

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 17, 2026

k8s-ci-robot requested review from kshalot and sohankunkerkar May 17, 2026 22:11

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 17, 2026

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 17, 2026

kevin85421 commented May 17, 2026

View reviewed changes

kevin85421 force-pushed the poc/rayservice-upgrade-quota-gate branch from 0bdc7b6 to 990a514 Compare May 17, 2026 22:23

kevin85421 commented May 18, 2026

View reviewed changes

This was referenced May 18, 2026

MultiKueue: Support Elastic RayService #11102

Open

[RayService][Kueue] Support top-level Spec.Suspend for zero-downtime upgrade ray-project/kuberay#4841

Merged

Future-Outlier mentioned this pull request May 19, 2026

[Bug] Redis cleanup job does not work with Kueue ray-project/kuberay#4773

Closed

2 tasks

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 19, 2026

kevin85421 force-pushed the poc/rayservice-upgrade-quota-gate branch from 990a514 to d405bb1 Compare May 30, 2026 21:43

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 30, 2026

kevin85421 commented Jun 7, 2026

View reviewed changes

Comment thread test/e2e/singlecluster/extended/kuberay_test.go

kevin85421 added 5 commits June 14, 2026 16:02

kevin85421 force-pushed the poc/rayservice-upgrade-quota-gate branch from 3b8d1dc to 7cf656b Compare June 14, 2026 23:02

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jun 14, 2026

kevin85421 added 5 commits June 14, 2026 16:19

docs: trim unsuspendAdmittedChildren comment to the essential behavior

a09b197

docs: drop owned-resource enumeration from Suspend comment

c8b9f4f

docs: trim Suspend comment to the essential behavior

47d208c

refactor: extract childRayClusterLabels helper for RayService child s…

9540ed6

…elector

docs: trim RunWithPodSetsInfo suspend comment

1828096

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 20, 2026

kevin85421 mentioned this pull request Aug 13, 2026

KEP-13243: MultiKueue worker-side Ray in-tree autoscaling for elastic jobs #13494

Open

Navigation Menu

Uh oh!

POC: gate RayService zero-downtime upgrade with workload slicing - #11264

POC: gate RayService zero-downtime upgrade with workload slicing#11264
kevin85421 wants to merge 10 commits into
kubernetes-sigs:mainfrom
kevin85421:poc/rayservice-upgrade-quota-gate

kevin85421 commented May 17, 2026 •

edited

Loading

Uh oh!

k8s-ci-robot commented May 17, 2026

Uh oh!

netlify Bot commented May 17, 2026 •

edited

Loading

Uh oh!

linux-foundation-easycla Bot commented May 17, 2026 •

edited

Loading

Uh oh!

k8s-ci-robot commented May 17, 2026

Uh oh!

kevin85421 May 17, 2026 •

edited

Loading

Uh oh!

kevin85421 May 18, 2026

Uh oh!

coderabbitai Bot commented Jun 7, 2026 •

edited

Loading

Review skipped

Uh oh!

Uh oh!

kevin85421 commented Jun 14, 2026

Uh oh!

k8s-ci-robot commented Jun 14, 2026

Uh oh!

kevin85421 commented Jun 15, 2026

Uh oh!

k8s-ci-robot commented Jun 20, 2026

Uh oh!

mimowo commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

kevin85421 commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design

Prerequisites for testing

Known limitations

Test plan

Uh oh!

k8s-ci-robot commented May 17, 2026

Uh oh!

netlify Bot commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Uh oh!

linux-foundation-easycla Bot commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented May 17, 2026

Uh oh!

kevin85421 May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevin85421 May 18, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Jun 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Uh oh!

kevin85421 commented Jun 14, 2026

Uh oh!

k8s-ci-robot commented Jun 14, 2026

Uh oh!

kevin85421 commented Jun 15, 2026

Uh oh!

k8s-ci-robot commented Jun 20, 2026

Uh oh!

mimowo commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kevin85421 commented May 17, 2026 •

edited

Loading

netlify Bot commented May 17, 2026 •

edited

Loading

linux-foundation-easycla Bot commented May 17, 2026 •

edited

Loading

kevin85421 May 17, 2026 •

edited

Loading

coderabbitai Bot commented Jun 7, 2026 •

edited

Loading